Transmission of files and streams between a sender and a recipient over a communications channel has been the subject of much literature. Preferably, a recipient desires to receive an exact copy of data transmitted over a channel by a sender with some level of certainty. Where the channel does not have perfect fidelity (which covers most all physically realizable systems), one concern is how to deal with data lost or garbled in transmission. Lost data (erasures) are often easier to deal with than corrupted data (errors) because the recipient cannot always tell when corrupted data is data received in error. Many error-correcting codes have been developed to correct for erasures and/or for errors. Typically, the particular code used is chosen based on some information about the infidelities of the channel through which the data is being transmitted and the nature of the data being transmitted. For example, where the channel is known to have long periods of infidelity, a burst error code might be best suited for that application. Where only short, infrequent errors are expected a simple parity code might be best.
Data transmission is straightforward when a transmitter and a receiver have all of the computing power and electrical power needed for communications and the channel between the transmitter and receiver is clean enough to allow for relatively error-free communications. The problem of data transmission becomes more difficult when the channel is in an adverse environment or the transmitter and/or receiver has limited capability.
One solution is the use of forward error correcting (FEC) techniques, wherein data is coded at the transmitter such that a receiver can recover from transmission erasures and errors. Where feasible, a reverse channel from the receiver to the transmitter allows for the receiver to communicate about errors to the transmitter, which can then adjust its transmission process accordingly. Often, however, a reverse channel is not available or feasible or is available only with limited capacity. For example, where the transmitter is transmitting to a large number of receivers, the transmitter might not be able to handle reverse channels from all those receivers. As another example, the communication channel may be a storage medium and thus the transmission of the data is forward through time and, unless someone invents a time travel machine that can go back in time, a reverse channel for this channel is infeasible. As a result, communication protocols often need to be designed without a reverse channel or with a limited capacity reverse channel and, as such, the transmitter may have to deal with widely varying channel conditions without a full view of those channel conditions.
The problem of data transmission between transmitters and receivers is made more difficult when the receivers need to be low-power, small devices that might be portable or mobile and need to receive data at high bandwidths. For example, a wireless network might be set up to deliver files or streams from a stationary transmitter to a large or indeterminate number of portable or mobile receivers either as a broadcast or multicast where the receivers are constrained in their computing power, memory size, available electrical power, antenna size, device size and other design constraints. Another example is in storage applications where the receiver retrieves data from a storage medium which exhibits infidelities in reproduction of the original data. Such receivers are often embedded with the storage medium itself in devices, for example disk drives, which are highly constrained in terms of computing power and electrical power.
In such a system, considerations to be addressed include having little or no reverse channel, limited memory, limited computing cycles, power, mobility and timing. Preferably, the design should minimize the amount of transmission time needed to deliver data to potentially a large population of receivers, where individual receivers and might be turned on and off at unpredictable times, move in and out of range, incur losses due to link errors, mobility, congestion forcing lower priority file or stream packets to be temporarily dropped, etc.
In the case of a packet protocol used for data transport over a channel that can lose packets, a file, stream or other block of data to be transmitted over a packet network is partitioned into equal size input symbols, encoding symbols the same size as the input symbols are generated from the input symbols using an FEC code, and the encoding symbols are placed and sent in packets. The “size” of a symbol can be measured in bits, whether or not the symbol is actually broken into a bit stream, where a symbol has a size of M bits when the symbol is selected from an alphabet of 2M symbols. In such a packet-based communication system, a packet oriented erasure FEC coding scheme might be suitable. A file transmission is called reliable if it allows the intended recipient to recover an exact copy of the original file even in the face of erasures in the network. A stream transmission is called reliable if it allows the intended recipient to recover an exact copy of each part of the stream in a timely manner even in the face of erasures in the network. Both file transmission and stream transmission can also be somewhat reliable, in the sense that some parts of the file or stream are not recoverable or for streaming if some parts of the stream are not recoverable in a timely fashion. Packet loss often occurs because sporadic congestion causes the buffering mechanism in a router to reach its capacity, forcing it to drop incoming packets. Protection against erasures during transport has been the subject of much study.
In the case of a protocol used for data transmission over a noisy channel that can corrupt bits, a block of data to be transmitted over a data transmission channel is partitioned into equal size input symbols, encoding symbols of the same size are generated from the input symbols and the encoding symbols are sent over the channel. For such a noisy channel the size of a symbol is typically one bit or a few bits, whether or not a symbol is actually broken into a bit stream. In such a communication system, a bit-stream oriented error-correction FEC coding scheme might be suitable. A data transmission is called reliable if it allows the intended recipient to recover an exact copy of the original block even in the face of errors (symbol corruption, either detected or undetected in the channel). The transmission can also be somewhat reliable, in the sense that some parts of the block may remain corrupted after recovery. Symbols are often corrupted by sporadic noise, periodic noise, interference, weak signal, blockages in the channel, and a variety of other causes. Protection against data corruption during transport has been the subject of much study.
Chain reaction codes are FEC codes that allow for generation of an arbitrary number of output symbols from the fixed input symbols of a file or stream. Sometimes, they are referred to as fountain or rateless FEC codes, since the code does not have an a priori fixed transmission rate. Chain reaction codes have many uses, including the generation of an arbitrary number of output symbols in an information additive way, as opposed to an information duplicative way, wherein the latter is where output symbols received by a receiver before being able to recover the input symbols duplicate already received information and thus do not provide useful information for recovering the input symbols. Novel techniques for generating, using and operating chain reaction codes are shown, for example, in Luby I, Luby II, Shokrollahi I and Shokrollahi II.
One property of the output symbols produced by a chain reaction encoder is that a receiver is able to recover the original file or block of the original stream as soon as enough output symbols have been received. Specifically, to recover the original K input symbols with a high probability, the receiver needs approximately K+A output symbols. The ratio A/K is called the “relative reception overhead.” The relative reception overhead depends on the number K of input symbols, and on the reliability of the decoder.
It is also known to use multi-stage chain reaction (“MSCR”) codes, such as those described in Shokrollahi I and/or II and developed by Digital Fountain, Inc. under the trade name “Raptor” codes. Multi-stage chain reaction codes are used, for example, in an encoder that receives input symbols from a source file or source stream, generates intermediate symbols from the input symbols and encodes the intermediate symbols using chain reaction codes. More particularly, a plurality of redundant symbols is generated from an ordered set of input symbols to be communicated. A plurality of output symbols are generated from a combined set of symbols including the input symbols and the redundant symbols, wherein the number of possible output symbols is much larger than the number of symbols in the combined set of symbols, wherein at least one output symbol is generated from more than one symbol in the combined set of symbols and from less than all of the symbols in the combined set of symbols, and such that the ordered set of input symbols can be regenerated to a desired degree of accuracy from any predetermined number, N, of the output symbols. It is also known to use the techniques described above to encode and decode systematic codes, in which the input symbols are includes amongst the possible output symbols of the code. This may be achieved as described in Shokrollahi II by first applying a transformation to the input symbols followed by the steps described above, said enhanced process resulting in the first output symbols generated by the code being equal to the input symbols. As will be clear to those of skill in the art of error and erasure coding, the techniques of Shokrollahi II may be applied directly to the codes described or suggested herein.
For some applications, other variations of codes might be more suitable or otherwise preferred.
The MSCR codes and chain reaction codes described above are extremely efficient in terms of their encoding and decoding complexity. One of the reasons for their efficiency is that the operations that are performed are linear operations over the field GF(2), i.e., the simple field over one bit where the operation of adding two field elements is simply the logical XOR operation, and the operation of multiplying two field elements is simply the logical AND operation. Generally these operations are performed over multiple bits concurrently, e.g., 32 bits at a time or 4 bytes at a time, and such operations are supported natively on all modern CPU processors. On the other hand, when used as erasure FEC codes, because the operations are over GF(2), it turns out that the chance that the receiver can decode all the input symbols goes down by at most approximately one-half for each additional symbol received beyond the first K, where K is the number of original input symbols. For example, if K+A encoding symbols are received then the chance that the recover process fails to recover the K original input symbols is at least 2−A. What would be a more desirable behavior is if the chance of decoding failure decreased more rapidly as a function of A.
There are other erasure and error-correcting FEC codes that operate over larger fields, for example Reed-Solomon codes that operate over GF(4), or over GF(8), or over GF(256), or more generally over GF(2L) for any L>1, and also LDPC codes that operate over larger fields. The advantage of such FEC codes is that, for example in the case of erasure FEC codes, the chance of decoding failure decreases much more rapidly as a function of A than FEC codes over GF(2). On the other hand, these FEC codes are typically much less efficient in terms of encoding and decoding complexity, and one of the primary reasons for that is because the operations over larger fields are much more complex and/or are not natively supported on modern CPUs, and the complexity typically grows as the field size grows. Thus, the FEC codes that operate over larger finite fields are often much slower or impractical compared to FEC codes that operate over GF(2).
Thus, what is needed are erasure and error-correcting FEC codes that are extremely efficient in terms of their encoding and decoding complexity with the property that the chance of decoding failure decreases very rapidly as a function of the number of symbols received beyond the minimal number needed by an ideal FEC code to recover the original input symbols.